Fast Text Processing for Information Retrieval

We describe an advanced text processing system for information retrieval from natural language document collections. We use both syntactic processing as well as statistical term clustering to obtain a representation of documents which would be more accurate than those obtained with more traditional key-word methods. A reliable top-down parser has been developed that allows for fast processing of large amounts of text, and for a precise identification of desired types of phrases for statistical analysis. Two statistical measures are computed: the measure of informational contribution of words in phrases, and the similarity measure between words. A P P R O X I M A T E P A R S I N G W I T H T T P T r p (Tagged Text Parser) is a top down English parser specifically designed for fast, reliable processing of large amounts of text. The parser operates on a tagged input, where each word has been marked with a tag indicating a syntactic category: a part of speech with selected morphological features such as number, tense, mode, case end degree) As an example, consider the following sentence from an article appearing in the Communications of the ACM: The binary number system often many advantages over a decimal representation for a high-performance, general-purpose computer. This sentence is tagged as follows (we show the best-tags option only; dt determiner, nn singular noun, nns plural noun, in preposition, jj adjective, vbz verb in present tense third person singular): [[the,dt],[binary,jj],[number,nn],[system,rm],[offers,vbz], [many,jj],[adventages,rms],[over, in], [a,dt],[decimal,jj], [representafion,nn], [for,in] ,[a, dt], [high_per formence,nn], [comS,comS],[general purpose,nn],[computer,nn],[perS,perS]] Tagging of the input text substantially reduces the search space of a top-down parser since it resolves many lexical ambiguities, such as singular verb vs. plural noun, past tense vs. past participle, or preposition vs. wh-determiner. Tagging also helps to reduce the number of parse structures that can be assigned to a sentence, decreases the demand for consulting of the dictionary, end simplifies dealing with unknown words. t At present we use the 35-tag Penn Treebank Tagset created at the University of Pennsylvania. Prior to parsing, the text is tagged automatically using a program supplied by Bolt Beranek and Newman. We wish to thank Ralph Weischedel and Marie Meeter of BBN for providing and assisting in the use of the tagger. T] 'P is based on the Linguistic String Grammar developed by Sager [8] and partially incorporated in the Proteus parser [3]. T I P is written in Quintus Prolog, and currently implements more than 400 grammar productions. The restriction component of the original LSP Grammar as well as the lamlxta-reduction based "semantics" of the Proteus implementation have been redesigned for the unification-hased environment. 2 TI 'P produces a regularized representation of each parsed sentence that reflects the sentence's logical structure. This representation may differ considerably from a standard parse tree, in that the constituents get moved around (e.g., de-passivization, de-dativization), and carrain noun phrases get transformed into equivalent clauses (denon'finalization). The aim is to produce a uniform representation across different paraphrases; for example, the phrase context-free language recognition or parsing is represented as shown below: [[verb, [or,[recognize,parse]]] [subject, anyone] [object, [np,[n,language], [edj,[context See]]]]]. The parser is equipped with a time-out mechanism that allows for fast closing of more difficult sub-constituents after a preset amount of time has elapsed without producing a parse. When the time-out option is turned on (which happens automatically during the parsing), the parser is permitted to skip portions of input to reach a starter terminal for the next constituent to be parsed, and closing the currently open one (or ones) with whatever partial representation has been generated thus far. The result is an approximate partial parse, which shows the overall structure of the sentence, from which some of the constituents may be missing. Since the time-out option can be regulated by setting an appropriate flag before the parsing starts, the parser may be tuned to reach an acceptable compromise between its speed and precision. The time-out mechanism is implemented using a straightforward parameter passing and is at present limited to only a subset of nonterminals used by the grammar. Suppose that X is such a nonterminal, and that it occurs on the right-hand side of a production S > X Y Z. The set of "starters" is computed for Y, which consists of the word tags that can occur as the left-most 2 See [10] for details.